Methods, systems, and manufacture for an e-commerce platform with augmented reality application for display of virtual objects

ABSTRACT

Disclosed are systems and methods for augmenting a customer image with at least one virtual object. One or more customer images depicting a customer environment are received from a customer computing device and 3D feature data for the customer images is determined. A virtual object is determined which corresponds to a desired merchant item. A positioning signal is received which corresponds to a desired location of virtual object in the customer environment, and a corresponding first location in each customer image at which to overlay the virtual object using the positioning signal is determined. An appropriate size and orientation of the virtual object is determined for each customer image based on corresponding 3D feature data, causing an overlay of an appropriately sized and oriented virtual object at the corresponding first location in customer images.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of and claims priority to U.S. patentSer. No. 16/229,369, filed Dec. 21, 2018, and entitled “E-COMMERCEPLATFORM WITH AUGMENTED REALITY APPLICATION FOR DISPLAY OF VIRTUALOBJECTS”. The content of the foregoing application is herebyincorporated by reference in its entirety for all purposes.

FIELD

The present disclosure relates generally to augmenting reality withvirtual objects, and in particular, augmenting one or more imagesdepicting a customer environment with a virtual object corresponding toa merchant item for sale.

BACKGROUND

Known augmented reality (AR) systems generally provide an interactiveexperience of a real-world environment where objects that reside in thereal-world are “augmented” by computer-generated perceptual information,such as across visual and/or auditory modalities. An overlaid visual canbe constructive (i.e. additive to the environment) or destructive(subtractive from the environment), and is typically seamlesslyinterwoven with the real world environment in a way that it is perceivedas an immersive aspect of the real environment. In this way, augmentedreality may alter a person's ongoing perception of the real worldenvironment. This is in comparison to virtual reality systems, whichgenerally completely replace a user's real world environment with asimulated one. Typically, augmented and virtual reality systems make useof headsets or special glasses, and require 3D cameras, generation ofcomplex 3D models, and advanced and computationally intensive processingtechniques such as object recognition or registration of images. Suchsystems are typically complex and expensive.

SUMMARY

Therefore, there is a need in the art for simplified augmented realitysystems and methods using generally available cameras and devices, andwhich do not require creation of a 3D model or computationally intensiveprocessing. Such simplified systems and methods may utilize simple andreadily available personal computing devices such as laptops, tablets,and mobile phones, may make use of readily available AR functionality,software and hardware, would not require intensive computationalprocessing, may occur in real-time or asynchronously, and would notrequire specialized equipment such as glasses or a headset foroperation. Such a simplified system may be used for the display ofvirtual objects in a customer environment, wherein the customerenvironment is provided and controlled by a customer, and wherein thevirtual objects correspond to merchant items for sale, allowing for acustomer, and in certain cases a merchant, to view and interact with thevirtual object in the customer's environment.

In embodiments, a computer-implemented method for augmenting an image ofa site with a virtual object may include receiving an image from a firstcomputing device, wherein the image depicts a potential site for thevirtual object and determining 3D feature data corresponding to aportion of the image. A selection of a virtual object for display may bereceived and a positioning signal corresponding to a desired location ofthe virtual object may be received. A first location in the image atwhich to overlay the virtual object using the positioning signal may bedetermined. An appropriate size and orientation of the virtual object inthe image based on the 3D feature data may be determined. An augmentedimage may be created for display on the first computing device byoverlaying an appropriately sized and oriented virtual object at thefirst location in the image.

In embodiments, the augmented image may be enabled to be displayed on asecond computing device. The method may further include receiving asecond positioning signal from the first computing device or the secondcomputing device which is indicative of a desired positioning of thevirtual object or a component part of the virtual object in theaugmented image. A second location in the augmented image of the site atwhich to overlay the virtual object or a component part of the virtualobject in response to the second positioning signal may be determined.An updated augmented image may be created in real time for display onthe first computing device or the second computing device by overlayingthe virtual object or a component part of the virtual object at thesecond location. In embodiments, the 3D feature data may include adetermined coordinate system relative to a camera of the first computingdevice. The 3D feature data may include location and size of planarsurfaces.

In embodiments, a computer-implemented method for augmenting a customerimage with at least one virtual object may include receiving one or morecustomer images from a customer computing device, wherein the one ormore customer images each depicts a customer environment and determining3D feature data corresponding to each of the one or more customerimages. A selection of a merchant item for display may be received, anda virtual object corresponding to the selection of a merchant item maybe determined. A positioning signal corresponding to a desired locationof virtual object in the customer environment may be received and afirst location in each of the one or more customer images at which tooverlay the virtual object using the positioning signal may bedetermined. An appropriate size and orientation of the virtual object ineach of the one or more customer images may be determined based oncorresponding 3D feature data, and the method may include causing anoverlay of an appropriately sized and oriented virtual object at thecorresponding first location in each of the one or more customer images.

In embodiments, causing the overlay includes transmitting at least oneof the appropriately sized and oriented virtual object and the firstlocation to the customer computing device. The customer computing deviceand the merchant computing device may be in a peer to peer relationship.

In embodiments, a re-positioning signal may be received from at leastone of the customer computing device, a merchant computing device, andanother computing device, wherein the repositioning signal is indicativeof a desired re-positioning of the virtual object. A second location maybe determined in one or more image frames of the customer image at whichto overlay the virtual object in response to the re-positioning signal.In embodiments, a method includes causing an updated overlay of anappropriately sized and oriented virtual object at the correspondingsecond location in each of the one or more customer images. Inembodiments, causing the updated overlay may include transmitting atleast one of the appropriately sized and oriented virtual object and thesecond location to the customer computing device. In embodiments, anupdated augmented image may be created for one or more customer imageswith the updated overlay for display on the customer computing deviceand/or on the merchant computing device. An augmented image for one ormore customer images may be created with the overlay, wherein theaugmented image is created upon receipt of one or more customer imagesfrom a customer computing device. In embodiments, an augmented image forthe one or more customer images may be created and displayed at adifferent time than the receipt of the one or more customer images froma customer computing device.

In embodiments, a computer-implemented method for augmenting an image ofa site with a virtual object may include capturing an image by a firstcomputing device, wherein the image depicts a potential site for avirtual object and determining, by the first computing device, 3Dfeature data corresponding to a portion of the image. The image and the3D feature data may be communicated to a second computing device,wherein the second computing device is in a peer to peer relationshipwith the first computing device. The second computing device maydetermine a first location in the image at which to overlay the virtualobject and an appropriate size and orientation of the virtual object inthe image based on the 3D feature data. An augmented image may becreated by the first or second computing device for display on the firstand/or second computing device by overlaying an appropriately sized andoriented virtual object at the first location in the image.

In embodiments, a system for augmenting an image of a site with avirtual object may include an interface module that is configured toreceive an image from a first computing device, wherein the imagedepicts a potential site for the virtual object, receive a virtualobject for display, and receive a positioning signal corresponding to adesired location of the virtual object. The system may also include acomputing module that is configured to determine 3D feature datacorresponding to a portion of a received image, determine a firstlocation in the image at which to overlay the virtual object using thepositioning signal, determine an appropriate size and orientation of thevirtual object in the image based on the 3D feature data, and cause anoverlay of an appropriately sized and oriented virtual object at thefirst location in the image.

In embodiments, the computing module is structured to transmit at leastone of the appropriately sized and oriented virtual object and the firstlocation to the customer computing device. In embodiments, the computingmodule is structured to create an augmented image for one or morecustomer images with the overlay for display on the customer computingdevice.

The computing module may be structured to create an augmented image fordisplay on the first computing device by overlaying an appropriatelysized and oriented virtual object at the first location in the image. Inembodiments, the interface module may receive a second positioningsignal from the first computing device or the second computing devicewhich is indicative of a desired positioning of the virtual object or acomponent part of the virtual object in the augmented image. Thecomputing module may determine a second location in the augmented imageof the site at which to overlay the virtual object or a component partof the virtual object in response to the second positioning signal, andmay create an updated augmented image in real time for display on one ofthe first computing device and the second computing device by overlayingthe virtual object or a component part of the virtual object at thesecond location. In embodiments, the 3D feature data may include adetermined coordinate system relative to a camera of the first computingdevice. The 3D feature data may include location and size of planarsurfaces.

In embodiments, a computer-implemented system for augmenting a customerimage with at least one virtual object may include an interface moduleenabled to receive one or more customer images from a customer computingdevice, wherein the one or more customer images each depicts a customerenvironment, receive a selection of a merchant item for display, receivea positioning signal corresponding to a desired location of virtualobject in the customer environment. The system may further include acomputing module that is configured to determine 3D feature datacorresponding to each of the one or more customer images, determine avirtual object corresponding to the selection of a merchant item,determine a first location in each of the one or more customer images atwhich to overlay the virtual object using the positioning signal,determine an appropriate size and orientation of the virtual object ineach of the one or more customer images based on corresponding 3Dfeature data, and cause an overlay of an appropriately sized andoriented virtual object at the first location in the image and/or createan augmented image for each of the one or more customer images fordisplay on the customer computing device by overlaying an appropriatelysized and oriented virtual objection at the corresponding first locationin each of the one or more customer images. In embodiments, one or moreaugmented images may be enabled for display on a merchant computingdevice. In embodiments, the customer computing device and the merchantcomputing device may be in a peer to peer relationship. The interfacemodule may receive a re-positioning signal from the customer computingdevice, the merchant computing device, or another computing device whichis indicative of a desired re-positioning of the virtual object, and thecomputing module may determine a second location in one or more imageframes of the customer image at which to overlay the virtual object inresponse to the re-positioning signal and may create one or more updatedaugmented images in real time by overlaying an appropriately sized andoriented virtual object at the second location. At least one augmentedimage may be created in real time with the receipt of one or morecustomer images from a customer computing device. The augmented imagefor each of the one or more customer images may be created and displayedat a different time than the receipt of the one or more customer imagesfrom a customer computing device.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 depicts an embodiment of an e-commerce platform.

FIG. 2 depicts an embodiment of a home page of an administrator.

FIG. 3 depicts an embodiment of an augmented reality application, suchas for overlaying a virtual object in an image.

FIGS. 4 a-b depict an example of a process for the exemplary applicationof FIG. 3 .

FIGS. 5 a-b depict another example of a process for the exemplaryapplication of FIG. 3 .

FIG. 6 depicts an exemplary set of augmented images showing a virtualvase on a coffee table from four different perspectives.

DETAILED DESCRIPTION

The present disclosure will now be described in detail by describingvarious illustrative, non-limiting embodiments thereof with reference tothe accompanying drawings and exhibits. The disclosure may, however, beembodied in many different forms and should not be construed as beinglimited to the illustrative embodiments set forth herein. Rather, theembodiments are provided so that this disclosure will be thorough andwill fully convey the concept of the disclosure to those skilled in theart.

With reference to FIG. 1 , an embodiment e-commerce platform 100 isdepicted for providing merchant products and services to customers.While the disclosure throughout contemplates using the apparatus,system, applications, and process disclosed to view, evaluate, andpurchase both products and services, for simplicity the descriptionherein will refer to products. All references to products throughoutthis disclosure should also be understood to be references to productsand/or services, including physical products, digital content, tickets,subscriptions, services to be provided, and the like.

While the disclosure throughout contemplates that a ‘merchant’ and a‘customer’ may be more than individuals, for simplicity the descriptionherein may generally refer to merchants and customers as such. Allreferences to merchants and customers throughout this disclosure shouldalso be understood to be references to groups of individuals, companies,corporations, computing entities, and the like, and may representfor-profit or not-for-profit exchange of products. Further, while thedisclosure throughout refers to ‘merchants’ and ‘customers’, anddescribes their roles as such, the e-commerce platform 100 should beunderstood to more generally support users in an e-commerce environment,and all references to merchants and customers throughout this disclosureshould also be understood to be references to users, such as where auser is a merchant-user (e.g., a seller, retailer, wholesaler, orprovider of products), a customer-user (e.g., a buyer, purchase agent,or user of products), a prospective user (e.g., a user browsing andevaluating products and not yet committed to a purchase, a userevaluating the e-commerce platform 100 for potential use in marketingand selling products, and the like), a service provider user (e.g., ashipping provider 112, a financial provider, and the like), a company orcorporate user (e.g., a company representative for purchase, sales, oruse of products; an enterprise user; a customer relations or customermanagement agent, and the like), an information technology user, acomputing entity user (e.g., a computing bot for purchase, sales, or useof products), and the like.

The e-commerce platform 100 may provide a centralized system forproviding merchants with online resources and facilities for managingtheir business. The facilities described herein may be deployed in partor in whole through a machine that executes computer software, modules,program codes, and/or instructions on one or more processors which maybe part of or external to the platform 100. Merchants may utilize thee-commerce platform 100 for managing commerce with customers, such as byimplementing an e-commerce experience with customers through an onlinestore 138, through applications 142A-B, through channels 110A-B, throughPOS devices 152 in physical locations (e.g., a physical storefront orother location such as through a kiosk, terminal, reader, printer, 3Dprinter, and the like), by managing their business through thee-commerce platform 100, and by interacting with customers through acommunications facility 129 of the e-commerce platform 100, or anycombination thereof. A merchant may utilize the e-commerce platform 100as a sole commerce presence with customers, or in conjunction with othermerchant commerce facilities, such as through a physical store (e.g.,‘brick-and-mortar’ retail stores), a merchant off-platform website 104(e.g., a commerce Internet website or other internet or web property orasset supported by or on behalf of the merchant separately from thee-commerce platform), and the like. However, even these ‘other’ merchantcommerce facilities may be incorporated into the e-commerce platform,such as where POS devices 152 in a physical store of a merchant arelinked into the e-commerce platform 100, where a merchant off-platformwebsite 104 is tied into the e-commerce platform 100, such as through‘buy buttons’ that link content from the merchant off platform website104 to the online store 138, and the like.

The online store 138 may represent a multitenant facility comprising aplurality of virtual storefronts. In embodiments, merchants may manageone or more storefronts in the online store 138, such as through amerchant device 102 (e.g., computer, laptop computer, mobile computingdevice, and the like), and offer products to customers through a numberof different channels 110A-B (e.g., an online store 138; a physicalstorefront through a POS device 152; electronic marketplace, through anelectronic buy button integrated into a website or social media channelsuch as on a social network, social media page, social media messagingsystem; and the like). A merchant may sell across channels 110A-B andthen manage their sales through the e-commerce platform 100, wherechannels 110A may be provided internal to the e-commerce platform 100 orfrom outside the e-commerce channel 110B. A merchant may sell in theirphysical retail store, at pop ups, through wholesale, over the phone,and the like, and then manage their sales through the e-commerceplatform 100. A merchant may employ all or any combination of these,such as maintaining a business through a physical storefront utilizingPOS devices 152, maintaining a virtual storefront through the onlinestore 138, and utilizing a communication facility 129 to leveragecustomer interactions and analytics 132 to improve the probability ofsales. Throughout this disclosure the terms online store 138 andstorefront may be used synonymously to refer to a merchant's onlinee-commerce offering presence through the e-commerce platform 100, wherean online store 138 may refer to the multitenant collection ofstorefronts supported by the e-commerce platform 100 (e.g., for aplurality of merchants) or to an individual merchant's storefront (e.g.,a merchant's online store).

In embodiments, a customer may interact through a customer device 150(e.g., computer, laptop computer, mobile computing device, and thelike), a POS device 152 (e.g., retail device, a kiosk, an automatedcheckout system, and the like), or any other commerce interface deviceknown in the art. The e-commerce platform 100 may enable merchants toreach customers through the online store 138, through POS devices 152 inphysical locations (e.g., a merchant's storefront or elsewhere), topromote commerce with customers through various features of the platformor stand-alone applications, dialog via electronic communicationfacility 129, and the like, providing a system for reaching customersand facilitating merchant services for the real or virtual pathwaysavailable for reaching and interacting with customers.

In embodiments, and as described further herein, the e-commerce platform100 may be implemented through a processing facility including aprocessor and a memory, the processing facility storing a set ofinstructions that, when executed, cause the e-commerce platform 100 toperform the e-commerce and support functions as described herein. Theprocessing facility may be part of a server, client, networkinfrastructure, mobile computing platform, cloud computing platform,stationary computing platform, or other computing platform, and provideelectronic connectivity and communications between and amongst theelectronic components of the e-commerce platform 100, merchant devices102, payment gateways 106, application developers, channels 110A-B,shipping providers 112, customer devices 150, point of sale devices 152,and the like. The e-commerce platform 100 may be implemented as a cloudcomputing service, a software as a service (SaaS), infrastructure as aservice (IaaS), platform as a service (PaaS), desktop as a Service(DaaS), managed software as a service (MSaaS), mobile backend as aservice (MBaaS), information technology management as a service(ITMaaS), and the like, such as in a software and delivery model inwhich software is licensed on a subscription basis and centrally hosted(e.g., accessed by users using a client (for example, a thin client) viaa web browser or other application, accessed through by POS devices, andthe like). In embodiments, elements of the e-commerce platform 100 maybe implemented to operate on various platforms and operating systems,such as iOS, Android, on the web, and the like (e.g., the administrator114 being implemented in multiple instances for a given online store foriOS, Android, and for the web, each with similar functionality). Inembodiments, features or applications of the platform may be implementedvia a peer to peer network, such as between a customer device and amerchant device.

In embodiments, the online store 138 may be served to a customer device150 through a webpage provided by a server of the e-commerce platform100. The server may receive a request for the webpage from a browser orother application installed on the customer device 150, where thebrowser (or other application) connects to the server through an IPAddress, the IP address obtained by translating a domain name. Inreturn, the server sends back the requested webpage. Webpages may bewritten in or include Hypertext Markup Language (HTML), templatelanguage, JavaScript, and the like, or any combination thereof. Forinstance, HTML is a computer language that describes static informationfor the webpage, such as the layout, format, and content of the webpage.Website designers and developers may use the template language to buildwebpages that combine static content, which is the same on multiplepages, and dynamic content, which changes from one page to the next. Atemplate language may make it possible to re-use the static elementsthat define the layout of a webpage, while dynamically populating thepage with data from an online store. The static elements may be writtenin HTML, and the dynamic elements written in the template language. Thetemplate language elements in a file may act as placeholders, such thatthe code in the file is compiled and sent to the customer device 150 andthen the template language is replaced by data from the online store138, such as when a theme is installed. The template and themes mayconsider tags, objects, and filters. The client device web browser (orother application) then renders the page accordingly.

In embodiments, online stores 138 may be served by the e-commerceplatform 100 to customers, where customers can browse and purchase thevarious products available (e.g., add them to a cart, purchaseimmediately through a buy-button, and the like). Online stores 138 maybe served to customers in a transparent fashion without customersnecessarily being aware that it is being provided through the e-commerceplatform 100 (rather than directly from the merchant). Merchants may usea merchant configurable domain name, a customizable HTML theme, and thelike, to customize their online store 138. Merchants may customize thelook and feel of their website through a theme system, such as wheremerchants can select and change the look and feel of their online store138 by changing their theme while having the same underlying product andbusiness data shown within the online store's product hierarchy. Themesmay be further customized through a theme editor, a design interfacethat enables users to customize their website's design with flexibility.Themes may also be customized using theme-specific settings that changeaspects, such as specific colors, fonts, and pre-built layout schemes.The online store may implement a content management system for websitecontent. Merchants may author blog posts or static pages and publishthem to their online store 138, such as through blogs, articles, and thelike, as well as configure navigation menus. Merchants may upload images(e.g., for products), video, content, data, and the like to thee-commerce platform 100, such as for storage by the system (e.g. as data134). In embodiments, the e-commerce platform 100 may provide functionsfor resizing images, associating an image with a product, adding andassociating text with an image, adding an image for a new productvariant, protecting images, and the like.

As described herein, the e-commerce platform 100 may provide merchantswith transactional facilities for products through a number of differentchannels 110A-B, including the online store 138, over the telephone, aswell as through physical POS devices 152 as described herein. Thee-commerce platform 100 may include business support services 116, anadministrator 114, and the like associated with running an on-linebusiness, such as providing a domain service 118 associated with theironline store, payment services 120 for facilitating transactions with acustomer, shipping services 122 for providing customer shipping optionsfor purchased products, risk and insurance services 124 associated withproduct protection and liability, merchant billing, and the like.Services 116 may be provided via the e-commerce platform 100 or inassociation with external facilities, such as through a payment gateway106 for payment processing, shipping providers 112 for expediting theshipment of products, and the like.

In embodiments, the e-commerce platform 100 may provide for integratedshipping services 122 (e.g., through an e-commerce platform shippingfacility or through a third-party shipping carrier), such as providingmerchants with real-time updates, tracking, automatic rate calculation,bulk order preparation, label printing, and the like.

FIG. 2 depicts a non-limiting embodiment for a home page of anadministrator 114, which may show information about daily tasks, astore's recent activity, and the next steps a merchant can take to buildtheir business. In embodiments, a merchant may log in to administrator114 via a merchant device 102 such as from a desktop computer or mobiledevice, and manage aspects of their online store 138, such as viewingthe online store's 138 recent activity, updating the online store's 138catalog, managing orders, recent visits activity, total orders activity,and the like. In embodiments, the merchant may be able to access thedifferent sections of administrator 114 by using the sidebar, such asshown on FIG. 2 . Sections of the administrator 114 may include variousinterfaces for accessing and managing core aspects of a merchant'sbusiness, including orders, products, customers, available reports anddiscounts. The administrator 114 may also include interfaces formanaging sales channels for a store including the online store, mobileapplication(s) made available to customers for accessing the store(Mobile App), POS devices, and/or a buy button. The administrator 114may also include interfaces for managing applications (Apps) installedon the merchant's account; settings applied to a merchant's online store138 and account. A merchant may use a search bar to find products,pages, or other information. Depending on the device 102 or softwareapplication the merchant is using, they may be enabled for differentfunctionality through the administrator 114. For instance, if a merchantlogs in to the administrator 114 from a browser, they may be able tomanage all aspects of their online store 138. If the merchant logs infrom their mobile device (e.g., via a mobile application), they may beable to view all or a subset of the aspects of their online store 138,such as viewing the online store's 138 recent activity, updating theonline store's 138 catalog, managing orders, and the like.

More detailed information about commerce and visitors to a merchant'sonline store 138 may be viewed through acquisition reports or metrics,such as displaying a sales summary for the merchant's overall business,specific sales and engagement data for active sales channels, and thelike. Reports may include, acquisition reports, behavior reports,customer reports, finance reports, marketing reports, sales reports,custom reports, and the like. The merchant may be able to view salesdata for different channels 110A-B from different periods of time (e.g.,days, weeks, months, and the like), such as by using drop-down menus. Anoverview dashboard may be provided for a merchant that wants a moredetailed view of the store's sales and engagement data. An activity feedin the home metrics section may be provided to illustrate an overview ofthe activity on the merchant's account. For example, by clicking on a‘view all recent activity’ dashboard button, the merchant may be able tosee a longer feed of recent activity on their account. A home page mayshow notifications about the merchant's online store 138, such as basedon account status, growth, recent customer activity, and the like.Notifications may be provided to assist a merchant with navigatingthrough a process, such as capturing a payment, marking an order asfulfilled, archiving an order that is complete, and the like.

The e-commerce platform 100 may provide for a communications facility129 and associated merchant interface for providing electroniccommunications and marketing, such as utilizing an electronic messagingaggregation facility for collecting and analyzing communicationinteractions between merchants, customers, merchant devices 102,customer devices 150, POS devices 152, and the like, to aggregate andanalyze the communications, such as for increasing the potential forproviding a sale of a product, and the like. For instance, a customermay have a question related to a product, which may produce a dialogbetween the customer and the merchant (or automated processor-basedagent representing the merchant), where the communications facility 129analyzes the interaction and provides analysis to the merchant on how toimprove the probability for a sale.

The e-commerce platform 100 may provide a financial facility 120 forsecure financial transactions with customers, such as through a securecard server environment. The e-commerce platform 100 may store creditcard information, such as in payment card industry data (PCI)environments (e.g., a card server), to reconcile financials, billmerchants, perform automated clearing house (ACH) transfers between ane-commerce platform 100 financial institution account and a merchant'sback account (e.g., when using capital), and the like. These systems mayhave Sarbanes-Oxley Act (SOX) compliance and a high level of diligencerequired in their development and operation. The financial facility 120may also provide merchants with financial support, such as through thelending of capital (e.g., lending funds, cash advances, and the like)and provision of insurance. In addition, the e-commerce platform 100 mayprovide for a set of marketing and partner services and control therelationship between the e-commerce platform 100 and partners. They alsomay connect and onboard new merchants with the e-commerce platform 100.These services may enable merchant growth by making it easier formerchants to work across the e-commerce platform 100. Through theseservices, merchants may be provided help facilities via the e-commerceplatform 100.

In embodiments, online store 138 may support a great number ofindependently administered storefronts and process a large volume oftransactional data on a daily basis for a variety of products.Transactional data may include customer contact information, billinginformation, shipping information, information on products purchased,information on services rendered, and any other information associatedwith business through the e-commerce platform 100. In embodiments, thee-commerce platform 100 may store this data in a data facility 134. Thetransactional data may be processed to produce analytics 132, which inturn may be provided to merchants or third-party commerce entities, suchas providing consumer trends, marketing and sales insights,recommendations for improving sales, evaluation of customer behaviors,marketing and sales modeling, trends in fraud, and the like, related toonline commerce, and provided through dashboard interfaces, throughreports, and the like. The e-commerce platform 100 may store informationabout business and merchant transactions, and the data facility 134 mayhave many ways of enhancing, contributing, refining, and extractingdata, where over time the collected data may enable improvements toaspects of the e-commerce platform 100.

Referring again to FIG. 1 , in embodiments the e-commerce platform 100may be configured with a commerce management engine 136 for contentmanagement, task automation and data management to enable support andservices to the plurality of online stores 138 (e.g., related toproducts, inventory, customers, orders, collaboration, suppliers,reports, financials, risk and fraud, and the like), but be extensiblethrough applications 142A-B that enable greater flexibility and customprocesses required for accommodating an ever-growing variety of merchantonline stores, POS devices, products, and services, where applications142A may be provided internal to the e-commerce platform 100 orapplications 142B from outside the e-commerce platform 100. Inembodiments, an application 142A may be provided by the same partyproviding the platform 100 or by a different party. In embodiments, anapplication 142B may be provided by the same party providing theplatform 100 or by a different party. The commerce management engine 136may be configured for flexibility and scalability through portioning(e.g., sharding) of functions and data, such as by customer identifier,order identifier, online store identifier, and the like. The commercemanagement engine 136 may accommodate store-specific business logic andin some embodiments, may incorporate the administrator 114 and/or theonline store 138.

The commerce management engine 136 includes base or “core” functions ofthe e-commerce platform 100, and as such, as described herein, not allfunctions supporting online stores 138 may be appropriate for inclusion.For instance, functions for inclusion into the commerce managementengine 136 may need to exceed a core functionality threshold throughwhich it may be determined that the function is core to a commerceexperience (e.g., common to a majority of online store activity, such asacross channels, administrator interfaces, merchant locations,industries, product types, and the like), is re-usable across onlinestores 138 (e.g., functions that can be re-used/modified across corefunctions), limited to the context of a single online store 138 at atime (e.g., implementing an online store ‘isolation principle’, wherecode should not be able to interact with multiple online stores 138 at atime, ensuring that online stores 138 cannot access each other's data),provide a transactional workload, and the like. Maintaining control ofwhat functions are implemented may enable the commerce management engine136 to remain responsive, as many required features are either serveddirectly by the commerce management engine 136 or enabled through aninterface 140A-B, such as by its extension through an applicationprogramming interface (API) connection to applications 142A-B andchannels 110A-B, where interfaces 140A may be provided to applications142A and/or channels 110A inside the e-commerce platform 100 or throughinterfaces 140B provided to applications 142B and/or channels 110Boutside the e-commerce platform 100. Generally, the platform 100 mayinclude interfaces 140A-B (which may be extensions, connectors, APIs,and the like) which facilitate connections to and communications withother platforms, systems, software, data sources, code and the like.Such interfaces 140A-B may be an interface 140A of the commercemanagement engine 136 or an interface 140B of the platform 100 moregenerally. If care is not given to restricting functionality in thecommerce management engine 136, responsiveness could be compromised,such as through infrastructure degradation through slow databases ornon-critical backend failures, through catastrophic infrastructurefailure such as with a data center going offline, through new code beingdeployed that takes longer to execute than expected, and the like. Toprevent or mitigate these situations, the commerce management engine 136may be configured to maintain responsiveness, such as throughconfiguration that utilizes timeouts, queues, back-pressure to preventdegradation, and the like.

Although isolating online store data is important to maintaining dataprivacy between online stores 138 and merchants, there may be reasonsfor collecting and using cross-store data, such as for example, with anorder risk assessment system or a platform payment facility, both ofwhich require information from multiple online stores 138 to performwell. In embodiments, rather than violating the isolation principle, itmay be preferred to move these components out of the commerce managementengine 136 and into their own infrastructure within the e-commerceplatform 100.

In embodiments, the e-commerce platform 100 may provide for a platformpayment facility 120, which is another example of a component thatutilizes data from the commerce management engine 136 but may be locatedoutside so as to not violate the isolation principle. The platformpayment facility 120 may allow customers interacting with online stores138 to have their payment information stored safely by the commercemanagement engine 136 such that they only have to enter it once. When acustomer visits a different online store 138, even if they've never beenthere before, the platform payment facility 120 may recall theirinformation to enable a more rapid and correct check out. This mayprovide a cross-platform network effect, where the e-commerce platform100 becomes more useful to its merchants as more merchants join, such asbecause there are more customers who checkout more often because of theease of use with respect to customer purchases. To maximize the effectof this network, payment information for a given customer may beretrievable from an online store's checkout, allowing information to bemade available globally across online stores 138. It would be difficultand error prone for each online store 138 to be able to connect to anyother online store 138 to retrieve the payment information stored there.As a result, the platform payment facility may be implemented externalto the commerce management engine 136.

For those functions that are not included within the commerce managementengine 136, applications 142A-B provide a way to add features to thee-commerce platform 100. Applications 142A-B may be able to access andmodify data on a merchant's online store 138, perform tasks through theadministrator 114, create new flows for a merchant through a userinterface (e.g., that is surfaced through extensions/API), and the like.Merchants may be enabled to discover and install applications 142A-Bthrough application search, recommendations, and support 128. Inembodiments, core products, core extension points, applications, and theadministrator 114 may be developed to work together. For instance,application extension points may be built inside the administrator 114so that core features may be extended by way of applications, which maydeliver functionality to a merchant through the extension.

In embodiments, applications 142A-B may deliver functionality to amerchant through the interface 140A-B, such as where an application142A-B is able to surface transaction data to a merchant (e.g., App:“Engine, surface my app data in mobile and web admin using the embeddedapp SDK”), and/or where the commerce management engine 136 is able toask the application to perform work on demand (Engine: “App, give me alocal tax calculation for this checkout”).

Applications 142A-B may support online stores 138 and channels 110A-B,provide for merchant support, integrate with other services, and thelike. Where the commerce management engine 136 may provide thefoundation of services to the online store 138, the applications 142A-Bmay provide a way for merchants to satisfy specific and sometimes uniqueneeds. Different merchants will have different needs, and so may benefitfrom different applications 142A-B. Applications 142A-B may be betterdiscovered through the e-commerce platform 100 through development of anapplication taxonomy (categories) that enable applications to be taggedaccording to a type of function it performs for a merchant; throughapplication data services that support searching, ranking, andrecommendation models; through application discovery interfaces such asan application store, home information cards, an application settingspage; and the like.

Applications 142A-B may be connected to the commerce management engine136 through an interface 140A-B, such as utilizing APIs to expose thefunctionality and data available through and within the commercemanagement engine 136 to the functionality of applications (e.g.,through REST, GraphQL, and the like). For instance, the e-commerceplatform 100 may provide API interfaces 140A-B to merchant andpartner-facing products and services, such as including applicationextensions, process flow services, developer-facing resources, and thelike. With customers more frequently using mobile devices for shopping,applications 142A-B related to mobile use may benefit from moreextensive use of APIs to support the related growing commerce traffic.The flexibility offered through use of applications and APIs (e.g., asoffered for application development) enable the e-commerce platform 100to better accommodate new and unique needs of merchants (and internaldevelopers through internal APIs) without requiring constant change tothe commerce management engine 136, thus providing merchants what theyneed when they need it. For instance, shipping services 122 may beintegrated with the commerce management engine 136 through a shipping orcarrier service API, thus enabling the e-commerce platform 100 toprovide shipping service functionality without directly impacting coderunning in the commerce management engine 136.

Many merchant problems may be solved by letting partners improve andextend merchant workflows through application development, such asproblems associated with back-office operations (merchant-facingapplications 142A-B) and in the online store 138 (customer-facingapplications 142A-B). As a part of doing business, many merchants willuse mobile and web related applications on a daily basis for back-officetasks (e.g., merchandising, inventory, discounts, fulfillment, and thelike) and online store tasks (e.g., applications related to their onlineshop, for flash-sales, new product offerings, and the like), whereapplications 142A-B, through extension/API 140A-B, help make productseasy to view and purchase in a fast growing marketplace. In embodiments,partners, application developers, internal applications facilities, andthe like, may be provided with a software development kit (SDK), such asthrough creating a frame within the administrator 114 that sandboxes anapplication interface. In embodiments, the administrator 114 may nothave control over nor be aware of what happens within the frame. The SDKmay be used in conjunction with a user interface kit to produceinterfaces that mimic the look and feel of the e-commerce platform 100,such as acting as an extension of the commerce management engine 136.

Applications 142A-B that utilize APIs may pull data on demand, but oftenthey also need to have data pushed when updates occur. Update events maybe implemented in a subscription model, such as for example, customercreation, product changes, or order cancelation. Update events mayprovide merchants with needed updates with respect to a changed state ofthe commerce management engine 136, such as for synchronizing a localdatabase, notifying an external integration partner, and the like.Update events may enable this functionality without having to poll thecommerce management engine 136 all the time to check for updates, suchas through an update event subscription. In embodiments, when a changerelated to an update event subscription occurs, the commerce managementengine 136 may post a request, such as to a predefined callback URL. Thebody of this request may contain a new state of the object and adescription of the action or event. Update event subscriptions may becreated manually, in the administrator facility 114, or automatically(e.g., via the API 140A-B). In embodiments, update events may be queuedand processed asynchronously from a state change that triggered them,which may produce an update event notification that is not distributedin real-time.

In embodiments, the e-commerce platform 100 may provide applicationsearch, recommendation and support 128. Application search,recommendation and support 128 may include developer products and toolsto aid in the development of applications, an application dashboard(e.g., to provide developers with a development interface, toadministrators for management of applications, to merchants forcustomization of applications, and the like), facilities for installingand providing permissions with respect to providing access to anapplication 142A-B (e.g., for public access, such as where criteria mustbe met before being installed, or for private use by a merchant),application searching to make it easy for a merchant to search forapplications 142A-B that satisfy a need for their online store 138,application recommendations to provide merchants with suggestions on howthey can improve the user experience through their online store 138, adescription of core application capabilities within the commercemanagement engine 136, and the like. These support facilities may beutilized by application development performed by any entity, includingthe merchant developing their own application 142A-B, a third-partydeveloper developing an application 142A-B (e.g., contracted by amerchant, developed on their own to offer to the public, contracted foruse in association with the e-commerce platform 100, and the like), oran application 142A or 142B being developed by internal personalresources associated with the e-commerce platform 100. In embodiments,applications 142A-B may be assigned an application identifier (ID), suchas for linking to an application (e.g., through an API), searching foran application, making application recommendations, and the like.

The commerce management engine 136 may include base functions of thee-commerce platform 100 and expose these functions through APIs 140A-Bto applications 142A-B. The APIs 140A-B may enable different types ofapplications built through application development. Applications 142A-Bmay be capable of satisfying a great variety of needs for merchants butmay be grouped roughly into three categories: customer-facingapplications, merchant-facing applications, integration applications,and the like. Customer-facing applications 142A-B may include onlinestore 138 or channels 110A-B that are places where merchants can listproducts, have them evaluated, and have them purchased (e.g., the onlinestore, applications for flash sales (e.g., merchant products or fromopportunistic sales opportunities from third-party sources), a mobilestore application, a social media channel, an application for providingwholesale purchasing, and the like). Merchant-facing applications 142A-Bmay include applications that allow the merchant to administer theironline store 138 (e.g., through applications related to the web orwebsite or to mobile devices), run their business (e.g., throughapplications related to POS devices), to grow their business (e.g.,through applications related to shipping (e.g., drop shipping), use ofautomated agents, use of process flow development and improvements), andthe like. Integration applications may include applications that provideuseful integrations that participate in the running of a business, suchas shipping providers 112 and payment gateways.

In embodiments, an application developer may use an application proxy tofetch data from an outside location and display it on the page of anonline store 138. Content on these proxy pages may be dynamic, capableof being updated, and the like. Application proxies may be useful fordisplaying image galleries, statistics, custom forms, and other kinds ofdynamic content. The core-application structure of the e-commerceplatform 100 may allow for an increasing number of merchant experiencesto be built in applications 142A-B so that the commerce managementengine 136 can remain focused on the more commonly utilized businesslogic of commerce.

The e-commerce platform 100 provides an online shopping experiencethrough a curated system architecture that enables merchants to connectwith customers in a flexible and transparent manner. A typical customerexperience may be better understood through an embodiment examplepurchase workflow, where the customer browses the merchant's products ona channel 110A-B, evaluates one or more products such as via anaugmented reality application, adds what they intend to buy to theircart, proceeds to checkout, and pays for the content of their cartresulting in the creation of an order for the merchant. The merchant maythen review and fulfill (or cancel) the order. The product is thendelivered to the customer. If the customer is not satisfied, they mightreturn the products to the merchant.

In an example embodiment, a customer may browse a merchant's products ona channel 110A-B. A channel 110A-B is a place where customers can view,evaluate, and buy products. In embodiments, channels 110A-B may bemodeled as applications 142A-B (a possible exception being the onlinestore 138, which is integrated within the commence management engine136). A merchandising component may allow merchants to describe whatthey want to sell and where they sell it. The association between aproduct and a channel may be modeled as a product publication andaccessed by channel applications, such as via a product listing API. Aproduct may have many options, like size and color, and many variantsthat expand the available options into specific combinations of all theoptions, like the variant that is extra-small and green, or the variantthat is size large and blue. Products may have at least one variant(e.g., a “default variant” is created for a product without anyoptions). To facilitate browsing and management, products may be groupedinto collections, provided product identifiers (e.g., stock keeping unit(SKU)) and the like. Collections of products may be built by eithermanually categorizing products into one (e.g., a custom collection), bybuilding rulesets for automatic classification (e.g., a smartcollection), and the like. Products may be viewed as 2D images, 3Dimages, rotating view images, through a virtual or augmented realityinterface or application, and the like.

In embodiments, the customer may add what they intend to buy to theircart (in an alternate embodiment, a product may be purchased directly,such as through a buy button as described herein). Customers may addproduct variants to their shopping cart. The shopping cart model may bechannel specific. The online store 138 cart may be composed of multiplecart line items, where each cart line item tracks the quantity for aproduct variant. Merchants may use cart scripts to offer specialpromotions to customers based on the content of their cart. Since addinga product to a cart does not imply any commitment from the customer orthe merchant, and the expected lifespan of a cart may be in the order ofminutes (not days), carts may be persisted to an ephemeral data store.

The customer then proceeds to checkout. A checkout component mayimplement a web checkout as a customer-facing order creation process. Acheckout API may be provided as a computer-facing order creation processused by some channel applications to create orders on behalf ofcustomers (e.g., for point of sale). Checkouts may be created from acart and record a customer's information such as email address, billing,and shipping details. On checkout, the merchant commits to pricing. Ifthe customer inputs their contact information but does not proceed topayment, the e-commerce platform 100 may provide an opportunity tore-engage the customer (e.g., in an abandoned checkout feature). Forthose reasons, checkouts can have much longer lifespans than carts(hours or even days) and are therefore persisted. Checkouts maycalculate taxes and shipping costs based on the customer's shippingaddress. Checkout may delegate the calculation of taxes to a taxcomponent and the calculation of shipping costs to a delivery component.A pricing component may enable merchants to create discount codes (e.g.,‘secret’ strings that when entered on the checkout apply new prices tothe items in the checkout). Discounts may be used by merchants toattract customers and assess the performance of marketing campaigns.Discounts and other custom price systems may be implemented on top ofthe same platform piece, such as through price rules (e.g., a set ofprerequisites that when met imply a set of entitlements). For instance,prerequisites may be items such as “the order subtotal is greater than$100” or “the shipping cost is under $10”, and entitlements may be itemssuch as “a 20% discount on the whole order” or “$10 off products X, Y,and Z”.

Customers then pay for the content of their cart resulting in thecreation of an order for the merchant. Channels 110A-B may use thecommerce management engine 136 to move money, currency or a store ofvalue (such as dollars or a cryptocurrency) to and from customers andmerchants. Communication with the various payment providers (e.g.,online payment systems, mobile payment systems, digital wallet, creditcard gateways, and the like) may be implemented within a paymentprocessing component. The actual interactions with the payment gateways106 may be provided through a card server environment. In embodiments,the payment gateway 106 may accept international payment, such asintegrating with leading international credit card processors. The cardserver environment may include a card server application, card sink,hosted fields, and the like. This environment may act as the securegatekeeper of the sensitive credit card information. In embodiments,most of the process may be orchestrated by a payment processing job. Thecommerce management engine 136 may support many other payment methods,such as through an offsite payment gateway 106 (e.g., where the customeris redirected to another website), manually (e.g., cash), online paymentmethods (e.g., online payment systems, mobile payment systems, digitalwallet, credit card gateways, and the like), gift cards, and the like.At the end of the checkout process, an order is created. An order is acontract of sale between the merchant and the customer where themerchant agrees to provide the goods and services listed on the orders(e.g., order line items, shipping line items, and the like) and thecustomer agrees to provide payment (including taxes). This process maybe modeled in a sales component. Channels 110A-B that do not rely oncommerce management engine 136 checkouts may use an order API to createorders. Once an order is created, an order confirmation notification maybe sent to the customer and an order placed notification sent to themerchant via a notification component. Inventory may be reserved when apayment processing job starts to avoid over-selling (e.g., merchants maycontrol this behavior from the inventory policy of each variant).Inventory reservation may have a short time span (minutes) and may needto be very fast and scalable to support flash sales (e.g., a discount orpromotion offered for a short time, such as targeting impulse buying).The reservation is released if the payment fails. When the paymentsucceeds, and an order is created, the reservation is converted into along-term inventory commitment allocated to a specific location. Aninventory component may record where variants are stocked, and tracksquantities for variants that have inventory tracking enabled. It maydecouple product variants (a customer facing concept representing thetemplate of a product listing) from inventory items (a merchant facingconcept that represent an item whose quantity and location is managed).An inventory level component may keep track of quantities that areavailable for sale, committed to an order or incoming from an inventorytransfer component (e.g., from a vendor).

The merchant may then review and fulfill (or cancel) the order. A reviewcomponent may implement a business process merchant's use to ensureorders are suitable for fulfillment before actually fulfilling them.Orders may be fraudulent, require verification (e.g., ID checking), havea payment method which requires the merchant to wait to make sure theywill receive their funds, and the like. Risks and recommendations may bepersisted in an order risk model. Order risks may be generated from afraud detection tool, submitted by a third-party through an order riskAPI, and the like. Before proceeding to fulfillment, the merchant mayneed to capture the payment information (e.g., credit card information)or wait to receive it (e.g., via a bank transfer, check, and the like)and mark the order as paid. The merchant may now prepare the productsfor delivery. In embodiments, this business process may be implementedby a fulfillment component. The fulfillment component may group the lineitems of the order into a logical fulfillment unit of work based on aninventory location and fulfillment service. The merchant may review,adjust the unit of work, and trigger the relevant fulfillment services,such as through a manual fulfillment service (e.g., at merchant managedlocations) used when the merchant picks and packs the products in a box,purchase a shipping label and input its tracking number, or just markthe item as fulfilled. A custom fulfillment service may send an email(e.g., a location that doesn't provide an API connection). An APIfulfillment service may trigger a third party, where the third-partyapplication creates a fulfillment record. A legacy fulfillment servicemay trigger a custom API call from the commerce management engine 136 toa third party (e.g., fulfillment by Amazon). A gift card fulfillmentservice may provision (e.g., generating a number) and activate a giftcard. Merchants may use an order printer application to print packingslips. The fulfillment process may be executed when the items are packedin the box and ready for shipping, shipped, tracked, delivered, verifiedas received by the customer, and the like.

If the customer is not satisfied, they may be able to return theproduct(s) to the merchant. The business process merchants may gothrough to “un-sell” an item may be implemented by a return component.Returns may consist of a variety of different actions, such as arestock, where the product that was sold actually comes back into thebusiness and is sellable again; a refund, where the money that wascollected from the customer is partially or fully returned; anaccounting adjustment noting how much money was refunded (e.g.,including if there was any restocking fees, or goods that weren'treturned and remain in the customer's hands); and the like. A return mayrepresent a change to the contract of sale (e.g., the order), and wherethe e-commerce platform 100 may make the merchant aware of complianceissues with respect to legal obligations (e.g., with respect to taxes).In embodiments, the e-commerce platform 100 may enable merchants to keeptrack of changes to the contract of sales over time, such as implementedthrough a sales model component (e.g., an append-only date-based ledgerthat records sale-related events that happened to an item).

In embodiments, the e-commerce platform 100 may provide an augmentedreality application for permitting customers to view various products asvirtual objects in a customer environment. In particular, the e-commerceplatform 100 may provide an option to users to interact with anaugmented reality application 300 (shown in FIG. 3 ), which allowscustomer users or merchant users to select a desired item for view as avirtual object in one or more images that are customer-supplied. Inembodiments, an augmented reality application may be provided to allowdirect interaction between a customer device and a merchant device. Forease of discussion, reference is made to a customer and a merchant,however, in embodiments it is understood that the disclosure applies toany two users (whether or not merchants or customers) and/or devices.

For example, a customer interested in a merchant product may be enabledto upload or share one or more images of the customer's environment,where, here and throughout the disclosure, the one or more images may bea single image, a set of separate images, or a set of video images. Theone or more images may be recorded, uploaded from storage, or capturedand shared in real time. As an example, a customer interested in a vasemay share one or more images of a set of shelves in the customer'sliving room, using the camera on the customer's mobile computing deviceto capture one or more images of the set of shelves, or to capture avideo of the shelves from various viewpoints, wherein the video may bestreamed in real time, and the augmented reality application may operateto augment the customer supplied one or more images with a virtualobject of the vase, such that an augmented one or more images may becreated depicting the vase on one of the shelves. The augmented one ormore images can be created and then displayed on the customer computingdevice (via a web browser or application interface or the like) andoptionally can also be similarly displayed on a computing device of themerchant. The customer or the merchant, via their respective computingdevices, may be able to provide control signals, such as touch screeninputs, mouse positioning, or the like, to position or reposition thevase, such as via a drag and drop action or other known manipulations.For example, the vase could be initially dragged and dropped at aposition on a selected shelf, then repositioned and/or reoriented on thesame or a different shelf or on another surface. The customer suppliedone or more images of the customer environment may depict the set ofshelves from various viewpoints in a room, allowing differentperspectives of the vase at various positions on the shelves to beviewed in an augmented image, set of images, or video.

In another example, a customer interested in a merchant product, such asa refrigerator, may be enabled to upload or share (possibly in realtime) one or more images of the customer's kitchen, and the augmentedreality application may operate to provide the customer with a virtualrefrigerator in place of an existing refrigerator or in a space providedfor a refrigerator. The augmented one or more images can be displayed onthe customer computing device and on a computing device of the merchant.The customer or the merchant, via their respective computing devices,may be able to provide control signals, such as touch screen inputs orthe like, to manipulate and move the virtual refrigerator or variouscomponents of the virtual refrigerator, such as to insure that there areno impediments to the refrigerator being moved into the desired space,or to illustrate various features of the refrigerator by opening andclosing various doors or drawers of the refrigerator. The customersupplied one or more images of the customer kitchen and the augmentedcustomer kitchen may take the form of a single image, a set of images,or a video of the entire kitchen including surrounding entrances, doors,counters, and other appliances, allowing for an evaluation of the size,fit, and features of the refrigerator.

In another example, multiple parties may collaborate to displaydifferent products for a customer. For example, a customer may wantdesign ideas for a room and a home furnishings merchant may partner withan interior designer to showcase various products, paint ideas,arrangements, and the like for a customer environment. The customer maybe enabled to upload or share (possibly in real time) one or more images(including video or a video stream) of the customer's dining room, andthe augmented reality application may enable collaboration with a homefurnishings merchant as well as one or more interior designers, such asa furniture consultant and an art consultant. For example, a furnitureconsultant and an art consultant could operate together to provide thecustomer with various design arrangements including a dining table,various dining chairs, a chandelier, a rug, a hutch, framed artwork,sculptures, wall paint colors, and the like. The consultants couldoperate to add different virtual objects at the same time orsequentially. The various design arrangements could be presented asaugmented images to be displayed on a customer computing device, on acomputing device of the merchant, and on a computing device of theinterior designer, either simultaneously or asynchronously. Theaugmented images may also be shared with another party associated withthe customer, such as a family member or friend, such as on a separatedevice for such party. The customer, other party, the merchant, or theone or more interior designers, via their respective computing devices,may be able to provide control signals, such as touch screen inputs orthe like, to change, manipulate, and/or move various componentspresented, simultaneously or asynchronously. In an example, a team ofinterior designers may work at the same time to position various homefurnishings in a customer environment and then present the variousarrangements to a customer, who may view these arrangements at a latertime. Another party associated with the customer may separately view thearrangements on his computing device at the same time or a differenttime than the customer, and both may be enabled to view the augmenteddesign arrangements from various angles when in the actual customerenvironment by providing one or more new customer supplied images,wherein the virtual objects will be sized, positioned, and orientedaccordingly with respect to a new image. The customer supplied one ormore images of the dining room and the augmented dining room may takethe form of a single image, a set of images, or a video of the entiredining room including surrounding entrances, windows, doors, and thelike, allowing for the showcasing and evaluation of various furnishingsand arrangements.

With such an augmented reality application, the customer would exercisecontrol over what portion of the environment to provide as backgroundfor the virtual object and could exercise control over a desiredposition/orientation of the virtual object. In addition, or in thealternative a merchant or other entity may exercise control over adesired position/orientation of the virtual object. The augmentedreality application would provide customers new ways to evaluate andinteract with various products and provide merchants the ability toshowcase various features of these products and inspire desire for theseproducts, in part by demonstrating suitability of a product for aparticular customer location.

FIG. 3 depicts an embodiment of an application 300 for overlaying avirtual object in one or more images. Computing and communicationcomponents for the application can exist at various locations such as ata customer computing device 310, at a merchant computing device 320, aseparate server, and/or at the e-commerce platform 100, and theapplication may exist as part of, or outside of, e-commerce platform100. Application components may be duplicated in various locations.Application components may include an interface module enabled toreceive one or more customer images from a customer computing device,wherein the one or more customer images each depicts a customerenvironment, receive a selection of a merchant item for display, receivea positioning or repositioning signal corresponding to a desiredlocation of virtual object in the customer environment. Applicationcomponents may include a computing module comprising a processing engineand storage to execute processing related to determining 3D featurepoints and 3D feature data by evaluating one or more customer suppliedimages and other data, retrieving, creating, and resizing a virtualobject corresponding to a selected item (such as a merchant product),creating an augmented image, set of images, or video such as byoverlaying the virtual object at a selected or determined location inone or more customer supplied images (which may be provided from storageor in real time, for example as a real time video stream), and moving,repositioning, orienting, reorienting, sizing, or resizing the virtualobject, or component parts of the virtual object, in one or more imagesof the customer-supplied images. In embodiments, 3D feature data mayinclude one or more of camera position, camera orientation (including,without limitation, rotation, angle, roll, gamma, yaw, alpha, pitch,beta and the like), position of any detected surfaces, objects orfeatures, the size and orientation of those surfaces, objects orfeatures, planes, AR-related metadata and the like. In embodiments, 3Dfeature data may be associated with an image or series of images, forexample captured at a rate equal to the frame rate of a video (such as60 frames per second). In embodiments, the augmented reality of one ormore images with the virtual object depicting a selected merchantproduct can be displayed on both a customer computing device and amerchant computing device (as well as other computing devices), and bothcustomer and merchant (as well as others) can be enabled to control themovement, sizing, and orientation of the virtual object in thecustomer-supplied one or more images. In this way, a 3D model of thecustomer environment is not necessary, and those portions of a customerenvironment are provided as desired by a customer, such as via a cameraon a customer computing device capturing one or more images of apotential site for a merchant product. In embodiments, a peer to peertype architecture can be used. In embodiments, screen mirroring can beused, such that a display at the customer computing device or othercomputing device is mirrored in the merchant computing device or anothercomputing device.

As illustrated in FIGS. 4 a-b , an exemplary process 400 for augmentingone or more customer supplied images with one or more virtual objectsdepicting one or more selected merchant products may include thefollowing steps. In embodiments, the e-commerce platform 100 may beconfigured to provide functionality via a graphical user interface forusers to interact with an augmented reality application 300. Forexample, at a step 410, the application receives a customer request foraugmented reality from a customer user, such as via a graphical userinterface. In the example, the customer user or a merchant may select adesired merchant product to be viewed as a virtual object, whichtriggers a request to the application. Additionally, or in thealternative, at a step 420, the application may present, via thegraphical user interface, instructions to the customer user to captureand/or upload one or more images or a video depicting a desired customerenvironment for the desired merchant product. The instructions mayinclude information regarding providing one or more dimensions for oneor more items in one or more images (e.g., the customer is instructed toprovide an indication of a dimension between shelves of the set ofshelves, or provide width, height, and depth dimensions of a cutoutspace in a kitchen into which a refrigerator may be inserted). Theinstructions may additionally request information regarding a desiredlocation of the merchant product to be represented by the virtual objectin an augmented one or more images. Other possibilities exist. Forexample, steps 410 and 420 may be omitted in other embodiments.

At a step 430, the augmented reality application 300 may receive acustomer-supplied image, set of images, or video (such as in acompressed or uncompressed state) comprising one or more imagesdepicting an environment of the customer. The application may alsoreceive data regarding a dimension between identified and marked itemsdepicted in the customer environment (such as a dimension betweenshelves or a size of a cutout space for a refrigerator), or theapplication may calculate a dimension based on the size of a productselected (i.e., the application assumes that a width dimension of thecutout space for a selected refrigerator corresponds to a widthdimension of the selected refrigerator for sale or otherwise calculatesthe dimensions such as based on a reference object or properties of thecamera capturing the images. The application may also receiveinformation regarding an initial desired placement of the merchantproduct or may determine an initial placement based on a nature or typeof the merchant product and an object recognition algorithm operating onone or more images of the received images (e.g., the initial placementof the virtual object representing a selected refrigerator is within adetermined cutout recognized in images of the kitchen).

At a step 440, a virtual object corresponding to the merchant product isretrieved or created. For example, 3D models corresponding to merchantproducts may be stored in a library in the e-commerce platform (orelsewhere) or multiple 2D images of the product from variousperspectives may be retrieved and manipulated to create an appropriatevirtual object in known ways.

At a step 450, the application may receive a positioning signal, such asfrom the customer computing device or from the merchant computingdevice, where the positioning signal is indicative of a desiredpositioning of the virtual object or a component part of the virtualobject in the customer environment, as represented in the one or morecustomer supplied images.

At a step 460, the application operates to determine an appropriatesize, orientation, and/or position of the virtual object with respect toone or more determined locations depicted in the customer-supplied oneor more images. This may occur by determining a correspondence betweentouch screen input signals of a user on a display showing thecustomer-supplied images and corresponding distances in one or morecorresponding images, such that a location at which the virtual objector a component part of the virtual object should be moved to or orientedwith respect to may be determined.

At a step 470, an augmented one or more images may be created in realtime by overlaying the virtual object or a component part of the virtualobject at a determined location, or a delta may be determinedcorresponding to amounts the virtual object should be moved in one ormore dimensions with respect to a previous location and/or orientation.New size information of the virtual object could also be provided. Thedelta information and/or additional sizing information can becommunicated back to the merchant and/or customer computing devices, andthe virtual object adjusted accordingly in displayed image frames at thecomputing devices. In embodiments, any one or a combination of the stepsdescribed herein, including with respect to FIGS. 4 a-b , may beperformed by any one or more of the devices and software elementsdescribed herein.

FIGS. 5 a-b depict an exemplary process 500 for augmenting customersupplied images with a virtual object, which is similar to exemplaryprocess 400 but which includes software configured for determining 3Dfeature data corresponding to one or more items in an image. Suchsoftware may be native to a computing device or additive thereto. Forexample, the software can be used in conjunction with computing deviceshaving a camera and other existing sensors (motion sensor, gyroscope,accelerometer, etc.) in order to define a coordinate system, and to usethe defined coordinate system to track changes of the position andorientation of the camera between different acquired images. In such amanner, each acquired image is associated with a corresponding positionand orientation of the camera in the defined coordinate system. Forexample, the software (hereafter also referred to as “AR software”) maydetermine a defined coordinate system (e.g., x, y, z cartesiancoordinates) having its origin (0, 0, 0) corresponding to where acomputing device camera for acquiring images is located when a firstimage is acquired. The AR software, in conjunction with sensor data,enables the tracking of a location and an orientation of the camerarelative to the defined coordinate system.

Further, the AR software may also determine the existence of planarsurfaces or other features for items in or aspects of the capturedimages. For example, such an AR software tool may exist on a customercomputing device and may allow for feature points such as edges to berecognized, such that a planar surface in an image (e.g., a shelf, atable, a rug, a counter, a wall, and the like) can be simply and easilydetermined. This 3D feature data relating to items in the image isuseful for calculating location and orientation of virtual objects withrespect to the defined coordinate system. This 3D feature data may alsobe used for recalibrating one image with respect to another, such aswhere a computing device is located in the customer environment andacquires a second image that needs to be oriented with respect to afirst image with the defined coordinate system and origin at aparticular location or is moved to a different physical location at adifferent point in time and needs to be oriented with respect to acoordinate system used at another point in time. The 3D feature data mayor may not be communicated to the merchant computing device or to thee-commerce platform. As described herein, components of the applicationusing the data from the AR software can be located at various locations,such as at the customer computing device, at the merchant computingdevice, at a remote server, or at any combinations of these locationsand the overall architecture may be peer-to-peer or involve a server orcloud computing resource.

The AR data acquired may include camera position and orientationrelative to origin of surfaces that are detected by the software as wellas the size and orientation of those detected surfaces. For example, ifa customer supplied image depicts a living room with a coffee table, theAR data may include dimensions of the table and orientation of itssurface relative to the determined coordinate system. If a customerwants to view an augmented image in the customer environment but using anew image, the AR software is enabled to identify 3D feature points inthe new image and line up the new image with respect to the definedcoordinate system of the previous image such as by using, at least inpart, 3D feature data of the previous image.

The e-commerce platform 100 may be configured to provide functionalityvia a graphical user interface for users to interact with an augmentedreality application 300. For example, at a step 510, the augmentedreality application may receive a customer request for augmented realityfrom a customer user via a graphical user interface such as a browser.In that example, a merchant may select, via a corresponding graphicaluser interface such as a browser, one or more desired merchant productsto be viewed as a virtual object, which triggers a request to theapplication. In other embodiments, the customer may select one or moredesired products to be viewed, or other parties could select one or moreproducts to be viewed as one or more virtual objects.

Additionally, or alternatively, at a step 520, the application maypresent, via the graphical user interface, instructions to the customeruser to capture and/or upload an image or set of images depicting adesired customer environment for the desired merchant product. Acustomer user may capture one or more discrete images using the cameraof the customer computing device, may acquire a set of images by takinga video of a customer site, and/or may use a captured video to selectone or more separate images of the customer site. The instructions mayinclude information regarding use of the AR software to acquire 3Dfeature data regarding planar surfaces or other items or features in theimages. The instructions may additionally request information regardinga desired position of the merchant product to be represented by thevirtual object in an augmented one or more images. Other possibilitiesexist. For example, steps 510 and/or 520 may be omitted in otherembodiments.

At a step 530, the customer-supplied image or set of images depicting anenvironment of the customer are processed by the AR software at thecustomer computing device or elsewhere (such as using a server, cloudcomputing resource or on a merchant device) in order to determine 3Dfeature data of identified items in the images. The AR software orapplication may also calculate dimensions in the images with respect tothe defined coordinate system. The application may also receiveinformation regarding an initial desired placement of the merchantproduct, or may determine an initial placement based on a nature or typeof the merchant product and an object recognition algorithm operating onone or more customer supplied images (e.g., the initial placement of thevirtual object representing a selected refrigerator is within adetermined cutout recognized in images of the kitchen). A desiredinitial placement may be provided by a merchant, by a customer, or byanother consultant (such as an interior designer) working with themerchant and/or customer.

At a step 540, a virtual object corresponding to the merchant product isretrieved or created. For example, 3D models corresponding to merchantproducts may be stored in a library in the e-commerce platform (orelsewhere) or multiple 2D images of the product from variousperspectives may be retrieved and manipulated to create an appropriatevirtual object in known ways.

At a step 550, the application may receive a positioning signal, such asfrom the customer computing device or from the merchant computing deviceor from another device, where the positioning signal is indicative of adesired position and/or size and/or orientation of the virtual object ora component part of the virtual object in the customer environment, asrepresented in the one or more customer supplied images.

At a step 560, the application operates to determine an appropriatesize, orientation, and position of the virtual object with respect toone or more determined locations depicted in the customer-supplied oneor more images and with respect to aspects of the 3D feature data. Thisstep takes into account the received positioning signal as well as the3D feature data in order to determine where to position the virtualobject on a surface or item in the image, as more fully explainedherein. Additionally, this step may determine a correspondence betweentouch screen input signals of a user on a display of a customer-suppliedimage and distances of items in the corresponding one or more images,such that a location at which the virtual object or a component part ofthe virtual object should be moved to or oriented with respect to may bedetermined. In embodiments, two or more users may add and/or position(via their respective device) different objects to an image during asame session. This may occur simultaneously or sequentially.

At a step 570, an augmented image or set of images may be created, inreal time or asynchronously, by overlaying the virtual object or acomponent part of the virtual object at a determined location of acustomer supplied image and displaying the augmented image or set ofimages on a customer computing device. In embodiments, a delta may bedetermined corresponding to amounts the virtual object should be movedin one or more dimensions with respect to a prior location in thedetermined coordinate system. The delta information can be communicatedback to the merchant and/or another user and/or to the customercomputing devices, and the virtual object adjusted accordingly indisplayed images at either or both of the computing devices. Inembodiments, an augmented image is created on a customer computingdevice and displayed on the customer computing device using differentmodules or components of the AR application. In embodiments, any one ora combination of the steps described herein, including with respect toFIGS. 5 a-b , may be performed by any one or more of the devices andsoftware elements described herein.

Additionally, more than one session may occur with respect to a givenimage, set of images or space, wherein each session may involvedifferent objects and/or different positions for those objects. Inembodiments, the application may have a toggle between modes enabledsuch that two or more different versions including a same set of 3Dfeature data can be maintained as separate versions or could be addedtogether. Information may be tagged as belonging to a particularsession. For example, an art consultant may present a version of adining room that includes framed artworks on the walls with the virtualartwork objects being tagged to a first session. A furniture consultantmay present a version of the dining room that includes a rug, a diningtable, and a set of chairs, wherein these items are tagged to a secondsession. In a first mode, the two versions are presented separately,while in a second mode, the two versions are additive by pulling in thetagged data from both sessions to present a cumulative view.

For example, in a scenario where a customer computing device acquiresone or more images and processes the one or more images to acquire the3D feature data, the images and the 3D feature data may be communicatedto the merchant computing device (or to a server of the platformcontrolling a website in communication with the merchant computingdevice) such that the merchant computing device or the server cancompute appropriate location information (i.e., size, position, andorientation) for a virtual object in the one or more images. Thelocation information of a virtual object may be determined with respectto the determined coordinate system of the camera, with such informationavailable for each of the images, and can be sent back to the customercomputing device such that an appropriate size, position, andorientation of the virtual object in each of the customer suppliedimages can be achieved. In embodiments, a merchant user may utilize abrowser to access a website of the platform, the website havingJavaScript to determine when a customer user has uploaded a customersupplied image. Such a website may use WebGL, which would allow for 3Drendering to occur in the merchant's browser in order to recreate acustomer scene based on a “virtual camera” having a same position andorientation as the customer computing device camera when a correspondingimage is captured. The 3D objects in the recreated scene would havematching size, position, and orientation as the detected items (such asplanar surfaces) in the corresponding image. The website may overlay therecreated virtual scene on top of the corresponding image. A merchantuser may drag virtual objects onto the scene. For example, the websitecan detect when the merchant user's mouse is on top of one of thedetected 3D planar surfaces and determine the corresponding coordinateson the surface in order to place the virtual object in the scene at thecorresponding location. The website may use JavaScript to send thecoordinates and type of object back to the customer computing device,wherein the application augments the one or more images to have thevirtual object in the desired location, and one or more images can bedisplayed, all at once or separately, on either or both displays of thecustomer computing device and the merchant computing device. Display andmanipulation of virtual objects may occur in real time or asynchronouslybetween a merchant user and a customer user.

If a customer user or other user wants to view an augmented image in thecustomer environment but is using a new image, the AR software may beenabled to identify 3D feature points in the new image and line up thenew image with respect to the defined coordinate system of the previousimage. For example, a customer or other user may open the AR applicationand take a second image of a customer environment from a location thatis different than a first image, such as a few feet away from the firstposition, or using a different computing device. The application may beenabled to obtain 3D feature points from the second image andessentially line these up with 3D feature points of the first image suchthat the defined coordinate system of the first image is oriented tothat of the second image. In this manner, the AR application may beenabled to appropriately, and in real time, position virtual objects inthe second image.

In another example, as shown in FIG. 6 , an exemplary display ofaugmented images (such as in a browser) may include a set of images. Asshown, a user has uploaded four or more images of a coffee table fromdifferent points of view. Each image has associated with itcorresponding 3D feature data, such as a corresponding camera positionand orientation and an identified surface corresponding to the top ofthe coffee table. A merchant user or other user, located outside of thecustomer environment and having received this data, may move the vase toa location on the coffee table. The location of the vase is replicatedin each of the views such that the other images also show a vase in thesame location but from different points of view, and each of theaugmented images may be displayed for the customer to view at the sametime or at a later time and for the merchant to also view. A customeruser can then reposition the virtual vase in one of the images displayedon the customer computing device, and the changed position of the vaseis replicated for the other images as moving the virtual vase in oneimage would operate to move the vase in each of the other images in acorresponding way. In embodiments, this may be done in an interactivemanner or asynchronously. In embodiments, only one image withcorresponding 3D feature data needs to be shared to achieve thefunctionality described herein.

In an embodiment of a use case, a prospective customer may take photosof their coffee table from different angles using an AR enabled device.Another user, an interior designer, may view the photos of the coffeetable in a browser and can drag virtual objects (such as 3D objects)into the photos and the products will appear in the images, synchronizedand simultaneously across all of the images, at the correct distance andperspective. The prospective customer may also see the virtual objectsappear in the images, but also in AR where they can walk around to seethe virtual objects in place from different angles. In anotherembodiment, the prospective customer may later place other virtualobjects into the pictures, but without necessarily having access to theactual space with the coffee table at the time, instead using the imagescaptured previously. This may be done using a browser extension. Inaddition, it may be possible to have multiple browser windows open, withthe changes reflected in real time on each browser window, allowing forreal time collaboration. In embodiments, it may be possible to conductcontests, for example where images of an empty living room (along withassociated 3D feature data) are shared, contestants can decorate thespace with virtual objects and share their creations with otherparticipants.

In embodiments, only a small amount of data corresponding to thecustomer acquired image or set of images may need to be transmittedbetween devices, reducing bandwidth requirements and speedingprocessing. In embodiments, complete or partial coordinate, positioningand orientation data for the virtual object may be transmitted. Inembodiments, the space in a set of images may be mapped on to a realcoordinate system and used to track the virtual object. In otherembodiments, the position of the virtual object may be tracked inrelation to another object or reference point in a set of images andchanges in the position of the virtual object may simply be tracked aschanges in position with respect to that object or reference point inrespect of the set of images, requiring less data to be transmitted thanin respect of complete or partial coordinates as described above. Inembodiments, when the position or aspects of the virtual object aremanipulated from a device not capturing the images, such device may onlyneed to transmit the change in position, as opposed to composing thevirtual object into the video stream for transmission, reducing theamount of data to be transmitted. In embodiments, the device notcapturing the images may simply transmit user inputs and such inputs aremapped on the device capturing the images or another device to representnative inputs on such device which are processed to manipulate thevirtual object, which may further reduce the data to be transmitted. Forexample, such data may include the instruction type (such as add anobject or move object), the identity of the object (such as blue vase)and the touch/screen coordinates for the object (such as X, Y), wheresuch coordinates are the location where the screen was touched and not3D coordinates. In embodiments, image data, such as from various viewpoints, may be transmitted instead of video data. In embodiments, 3Dfeature data may be transmitted along with the corresponding image, setof images or video. In embodiments, transmitting an image of a spacealong with corresponding 3D feature data may be less bandwidth intensivethan transmitting a video of the space without corresponding 3D featuredata.

In embodiments, the disclosure herein provides for improved securitysince the customer maintains control over the aspects of the spacevisible to the merchant (such as due to the fact that the merchant isnot provided with a full 3D model of the space and can only view theaspects shown by the customer). In embodiments, the disclosure hereinprovides for improved security since the customer does not need to grantthe merchant access to the customer device for the merchant tomanipulate virtual objects in the space. In embodiments, the customermay create a video of a space, but then select only certain images fromthat video and share those, along with the corresponding 3D featuredata. This allows the customer to maintain control over the informationshared and also reduces the bandwidth, processing and other resourcesrequired.

In embodiments, the interactions between the customer and the merchantmay be synchronous (such as through a real time set of images, which maybe a video) or asynchronous (such as through a recorded set of images,which may be a video). In embodiments, a customer may share a recordedor stored one or more images of a space with a merchant. The merchantmay position one or more virtual objects in the space using the recordedor stored one or more images. When the customer views the recorded oneor more images, the customer may be able to see the virtual objectspositioned by the merchant. When the customer views the real space usingthe customer's mobile device the customer (such as through use of theaugmented reality application 300) may be able to see the virtualobjects in a real time image or camera feed viewed on the customerdevice or shared with another device. In embodiments, these asynchronousinteractions may be helpful where a user at a store stores or recordsone or more images of a shelf layout which is shared with a merchantwishing to sell products in such store. The merchant can use the storedor recorded video to position virtual products on the selves. Using theaugmented reality application 300 the user at the store may later viewthe shelves in real time using their mobile device and see the virtualproducts placed on such shelves by the merchant, such products placedusing the recorded one or more images. This may allow the user in thestore to place corresponding real objects on the shelves based onmirroring the placement of the virtual objects.

In embodiments, the virtual object may exist in a database or otherrepository of virtual objects. In embodiments, the virtual object may becreated from a photograph or video of a real object created using amobile or other device.

While the disclosure throughout generally contemplates providingaugmented reality images, sets of images and/or video with a virtualobject of a merchant product in a customer-supplied image or sets orseries of images (such as video), such systems and methods can begeneralized to other use cases. In addition, while the disclosurethroughout generally contemplates one or more products from a merchant,the disclosure is also applicable to services from a merchant, such asinstallation, remodeling, painting, cleaning and other services,including services which may alter the space (including, withoutlimitation, a surface of the space) shown in one or more images, andwhere the experience can show the space before and after performance ofa service.

The terms transmission, communication, receiving and the like usedherein, such as in relation to images, video, video streams, data(including 3D feature data) and the like, may refer to transmission,communication, receiving and the like within a given device (such asbetween modules, software components or hardware components of or on thedevice), within software on a given device or may be between devices,such as over a network.

The methods and systems described herein may be deployed in part or inwhole through a machine that executes computer software, program codes,and/or instructions on a processor. The processor may be part of aserver, cloud server, client, network infrastructure, mobile computingplatform, stationary computing platform, or other computing platform. Aprocessor may be any kind of computational or processing device capableof executing program instructions, codes, binary instructions and thelike. The processor may be or include a signal processor, digitalprocessor, embedded processor, microprocessor or any variant such as aco-processor (math co-processor, graphic co-processor, communicationco-processor and the like) and the like that may directly or indirectlyfacilitate execution of program code or program instructions storedthereon. In addition, the processor may enable execution of multipleprograms, threads, and codes. The threads may be executed simultaneouslyto enhance the performance of the processor and to facilitatesimultaneous operations of the application. By way of implementation,methods, program codes, program instructions and the like describedherein may be implemented in one or more thread. The thread may spawnother threads that may have assigned priorities associated with them;the processor may execute these threads based on priority or any otherorder based on instructions provided in the program code. The processormay include memory that stores methods, codes, instructions and programsas described herein and elsewhere. The processor may access a storagemedium through an interface that may store methods, codes, andinstructions as described herein and elsewhere. The storage mediumassociated with the processor for storing methods, programs, codes,program instructions or other type of instructions capable of beingexecuted by the computing or processing device may include but may notbe limited to one or more of a CD-ROM, DVD, memory, hard disk, flashdrive, RAM, ROM, cache and the like.

A processor may include one or more cores that may enhance speed andperformance of a multiprocessor. In embodiments, the process may be adual core processor, quad core processors, other chip-levelmultiprocessor and the like that combine two or more independent cores(called a die).

The methods and systems described herein may be deployed in part or inwhole through a machine that executes computer software on a server,cloud server, client, firewall, gateway, hub, router, or other suchcomputer and/or networking hardware. The software program may beassociated with a server that may include a file server, print server,domain server, internet server, intranet server and other variants suchas secondary server, host server, distributed server and the like. Theserver may include one or more of memories, processors, computerreadable media, storage media, ports (physical and virtual),communication devices, and interfaces capable of accessing otherservers, clients, machines, and devices through a wired or a wirelessmedium, and the like. The methods, programs or codes as described hereinand elsewhere may be executed by the server. In addition, other devicesrequired for execution of methods as described in this application maybe considered as a part of the infrastructure associated with theserver.

The server may provide an interface to other devices including, withoutlimitation, clients, other servers, printers, database servers, printservers, file servers, communication servers, distributed servers andthe like. Additionally, this coupling and/or connection may facilitateremote execution of program across the network. The networking of someor all of these devices may facilitate parallel processing of a programor method at one or more location without deviating from the scope ofthe disclosure. In addition, any of the devices attached to the serverthrough an interface may include at least one storage medium capable ofstoring methods, programs, code and/or instructions. A centralrepository may provide program instructions to be executed on differentdevices. In this implementation, the remote repository may act as astorage medium for program code, instructions, and programs.

The software program may be associated with a client that may include afile client, print client, domain client, internet client, intranetclient and other variants such as secondary client, host client,distributed client and the like. The client may include one or more ofmemories, processors, computer readable media, storage media, ports(physical and virtual), communication devices, and interfaces capable ofaccessing other clients, servers, machines, and devices through a wiredor a wireless medium, and the like. The methods, programs or codes asdescribed herein and elsewhere may be executed by the client. Inaddition, other devices required for execution of methods as describedin this application may be considered as a part of the infrastructureassociated with the client.

The client may provide an interface to other devices including, withoutlimitation, servers, other clients, printers, database servers, printservers, file servers, communication servers, distributed servers andthe like. Additionally, this coupling and/or connection may facilitateremote execution of program across the network. The networking of someor all of these devices may facilitate parallel processing of a programor method at one or more location without deviating from the scope ofthe disclosure. In addition, any of the devices attached to the clientthrough an interface may include at least one storage medium capable ofstoring methods, programs, applications, code and/or instructions. Acentral repository may provide program instructions to be executed ondifferent devices. In this implementation, the remote repository may actas a storage medium for program code, instructions, and programs.

The methods and systems described herein may be deployed in part or inwhole through network infrastructures. The network infrastructure mayinclude elements such as computing devices, servers, routers, hubs,firewalls, clients, personal computers, communication devices, routingdevices and other active and passive devices, modules and/or componentsas known in the art. The computing and/or non-computing device(s)associated with the network infrastructure may include, apart from othercomponents, a storage medium such as flash memory, buffer, stack, RAM,ROM and the like. The processes, methods, program codes, instructionsdescribed herein and elsewhere may be executed by one or more of thenetwork infrastructural elements.

The methods, program codes, and instructions described herein andelsewhere may be implemented in different devices which may operate inwired or wireless networks. Examples of wireless networks include 4^(th)Generation (4G) networks (e.g. Long Term Evolution (LTE)) or 5^(th)Generation (5G) networks, as well as non-cellular networks such asWireless Local Area Networks (WLANs). However, the principles describedtherein may equally apply to other types of networks.

The operations, methods, programs codes, and instructions describedherein and elsewhere may be implemented on or through mobile devices.The mobile devices may include navigation devices, cell phones, mobilephones, mobile personal digital assistants, laptops, palmtops, netbooks,pagers, electronic books readers, music players and the like. Thesedevices may include, apart from other components, a storage medium suchas a flash memory, buffer, RAM, ROM and one or more computing devices.The computing devices associated with mobile devices may be enabled toexecute program codes, methods, and instructions stored thereon.Alternatively, the mobile devices may be configured to executeinstructions in collaboration with other devices. The mobile devices maycommunicate with base stations interfaced with servers and configured toexecute program codes. The mobile devices may communicate on a peer topeer network, mesh network, or other communications network. The programcode may be stored on the storage medium associated with the server andexecuted by a computing device embedded within the server. The basestation may include a computing device and a storage medium. The storagedevice may store program codes and instructions executed by thecomputing devices associated with the base station.

The computer software, program codes, and/or instructions may be storedand/or accessed on machine readable media that may include: computercomponents, devices, and recording media that retain digital data usedfor computing for some interval of time; semiconductor storage known asrandom access memory (RAM); mass storage typically for more permanentstorage, such as optical discs, forms of magnetic storage like harddisks, tapes, drums, cards and other types; processor registers, cachememory, volatile memory, non-volatile memory; optical storage such asCD, DVD; removable media such as flash memory (e.g. USB sticks or keys),floppy disks, magnetic tape, paper tape, punch cards, standalone RAMdisks, Zip drives, removable mass storage, off-line, and the like; othercomputer memory such as dynamic memory, static memory, read/writestorage, mutable storage, read only, random access, sequential access,location addressable, file addressable, content addressable, networkattached storage, storage area network, bar codes, magnetic ink, and thelike.

The methods and systems described herein may transform physical and/oror intangible items from one state to another. The methods and systemsdescribed herein may also transform data representing physical and/orintangible items from one state to another, such as from usage data to anormalized usage dataset.

The elements described and depicted herein, including in flow charts andblock diagrams throughout the figures, imply logical boundaries betweenthe elements. However, according to software or hardware engineeringpractices, the depicted elements and the functions thereof may beimplemented on machines through computer executable media having aprocessor capable of executing program instructions stored thereon as amonolithic software structure, as standalone software modules, or asmodules that employ external routines, code, services, and so forth, orany combination of these, and all such implementations may be within thescope of the present disclosure. Examples of such machines may include,but may not be limited to, personal digital assistants, laptops,personal computers, mobile phones, other handheld computing devices,medical equipment, wired or wireless communication devices, transducers,chips, calculators, satellites, tablet PCs, electronic books, gadgets,electronic devices, devices having artificial intelligence, computingdevices, networking equipment, servers, routers and the like.Furthermore, the elements depicted in the flow chart and block diagramsor any other logical component may be implemented on a machine capableof executing program instructions. Thus, while the foregoing drawingsand descriptions set forth functional aspects of the disclosed systems,no particular arrangement of software for implementing these functionalaspects should be inferred from these descriptions unless explicitlystated or otherwise clear from the context. Similarly, it will beappreciated that the various steps identified and described above may bevaried, and that the order of steps may be adapted to particularapplications of the techniques disclosed herein. All such variations andmodifications are intended to fall within the scope of this disclosure.As such, the depiction and/or description of an order for various stepsshould not be understood to require a particular order of execution forthose steps, unless required by a particular application, or explicitlystated or otherwise clear from the context.

The methods and/or processes described above, and steps thereof, may berealized in hardware, software or any combination of hardware andsoftware suitable for a particular application. The hardware may includea general-purpose computer and/or dedicated computing device or specificcomputing device or particular aspect or component of a specificcomputing device. The processes may be realized in one or moremicroprocessors, microcontrollers, embedded microcontrollers,programmable digital signal processors or other programmable device,along with internal and/or external memory. The processes may also, orinstead, be embodied in an application specific integrated circuit, aprogrammable gate array, programmable array logic, or any other deviceor combination of devices that may be configured to process electronicsignals. It will further be appreciated that one or more of theprocesses may be realized as a computer executable code capable of beingexecuted on a machine readable medium.

The computer executable code may be created using a structuredprogramming language such as C, an object oriented programming languagesuch as C++, or any other high-level or low-level programming language(including assembly languages, hardware description languages, anddatabase programming languages and technologies) that may be stored,compiled or interpreted to run on one of the above devices, as well asheterogeneous combinations of processors, processor architectures, orcombinations of different hardware and software, or any other machinecapable of executing program instructions.

Thus, in one aspect, each method described above, and combinationsthereof may be embodied in computer executable code that, when executingon one or more computing devices, performs the steps thereof. In anotheraspect, the methods may be embodied in systems that perform the stepsthereof and may be distributed across devices in a number of ways, orall of the functionality may be integrated into a dedicated, standalonedevice or other hardware. In another aspect, the means for performingthe steps associated with the processes described above may include anyof the hardware and/or software described above. All such permutationsand combinations are intended to fall within the scope of the presentdisclosure.

What is claimed is:
 1. A computer-implemented method for augmenting acustomer image with at least one virtual object, the method comprising:receiving a plurality of customer images from a first computing device,wherein each of the plurality of customer images depicts a respectivedifferent 2D view of a customer environment; determining 3D feature datacorresponding to each of the plurality of customer images; receiving aselection of a merchant item for display; determining a virtual objectcorresponding to the selection of a merchant item; receiving from thefirst computing device a positioning signal corresponding to a desiredlocation of the virtual object in a particular 2D view of the customerenvironment; determining a corresponding first location in each of theplurality of customer images at which to overlay the virtual objectbased on the positioning signal, the corresponding first locationcorresponding to the desired location in each respective different 2Dview of the customer environment; determining a size and orientation ofthe virtual object in each of the plurality of customer images based oncorresponding 3D feature data; causing an overlay of the virtual object,having the determined size and orientation, at the corresponding firstlocation in each of the plurality of customer images to create arespective plurality of augmented images for display on a secondcomputing device, wherein the second computing device is a separatedevice from the first computing device; receiving a re-positioningsignal indicative of a desired re-positioned location of the virtualobject in the particular 2D view or a different 2D view of the customerenvironment; determining a corresponding second location in each of theplurality of customer images at which to overlay the virtual objectbased on the re-positioning signal, the corresponding second locationcorresponding to the desired re-positioned location in each respectivedifferent 2D view of the customer environment; and updating the overlayof the virtual object to be positioned at the corresponding secondlocation in each of the plurality of customer images.
 2. The method ofclaim 1, wherein causing the overlay includes transmitting at least oneof the virtual object having the determined size and orientation or thecorresponding first location to the first computing device.
 3. Themethod of claim 1, wherein the plurality of customer images receivedeach includes the 3D feature data.
 4. The method of claim 1, wherein the3D feature data includes a determined coordinate system relative to acamera of the first computing device.
 5. The method of claim 1, whereinthe 3D feature data includes location and size of planar surfaces. 6.The method of claim 1, wherein updating the overlay includestransmitting at least one of the virtual object having an updated sizeand orientation or the corresponding second location to the firstcomputing device.
 7. The method of claim 1, further comprising creatingan updated augmented image for one or more of the plurality of customerimages with the updated overlay for display on the first computingdevice.
 8. The method of claim 1, further comprising creating an updatedaugmented image for one or more of the plurality of customer images withthe updated overlay for display on the second computing device.
 9. Themethod of claim 1, further comprising creating an augmented image forone or more of the plurality of customer images with the overlay. 10.The method of claim 9, wherein the augmented image for the one or moreof the plurality of customer images is created and displayed at adifferent time than the receipt of the plurality of customer images fromthe first computing device.
 11. A computer system comprising: aprocessor; a memory coupled to the processor, the memory storinginstructions which, when executed by the processor, cause the system to:receive a plurality of customer images from a first computing device,wherein each of the plurality of customer images depicts a respectivedifferent 2D view of a customer environment; determine 3D feature datacorresponding to each of the plurality of customer images; receive aselection of a merchant item for display; determine a virtual objectcorresponding to the selection of a merchant item; receive from thefirst computing device a positioning signal corresponding to a desiredlocation of the virtual object in a particular 2D view of the customerenvironment; determine a corresponding first location in each of theplurality of customer images at which to overlay the virtual objectbased on the positioning signal, the corresponding first locationcorresponding to the desired location in each respective different 2Dview of the customer environment; determine a size and orientation ofthe virtual object in each of the plurality of customer images based oncorresponding 3D feature data; cause an overlay of the virtual object,having the determined size and orientation, at the corresponding firstlocation in each of the plurality of customer images to create arespective plurality of augmented images for display on a secondcomputing device, wherein the second computing device is a separatedevice from the first computing device; receive a re-positioning signalindicative of a desired re-positioned location of the virtual object inthe particular 2D view or a different 2D view of the customerenvironment; determine a corresponding second location in each of theplurality of customer images at which to overlay the virtual objectbased on the re-positioning signal, the corresponding second locationcorresponding to the desired re-positioned location in each respectivedifferent 2D view of the customer environment; and update the overlay ofthe virtual object to be positioned at the corresponding second locationin each of the plurality of customer images.
 12. The system of claim 11,wherein causing the overlay includes transmitting at least one of thevirtual object having the determined size and orientation or thecorresponding first location to the first computing device.
 13. Thesystem of claim 11, wherein the plurality of customer images receivedeach includes the 3D feature data.
 14. The system of claim 11, whereinthe 3D feature data includes a determined coordinate system relative toa camera of the first computing device.
 15. The system of claim 11,wherein the 3D feature data includes location and size of planarsurfaces.
 16. The system of claim 11, wherein updating the overlayincludes transmitting at least one of the virtual object having anupdated size and orientation or the corresponding second location to thefirst computing device.
 17. The system of claim 11, wherein theinstructions further cause the system to create updated augmented imagefor one or more of the plurality of customer images with the updatedoverlay for display on the first computing device.
 18. The system ofclaim 11, in the instructions further cause the system to create anupdated augmented image for one or more of the plurality of customerimages with the updated overlay for display on the second computingdevice.
 19. The system of claim 11, in the instructions further causethe system to create an augmented image for one or more of the pluralityof customer images with the overlay.
 20. A non-transitorycomputer-readable medium containing instructions which, when executed bya processor, cause the processor to: receive a plurality of customerimages from a first computing device, wherein each of the plurality ofcustomer images depicts a respective different 2D view of a customerenvironment; determine 3D feature data corresponding to each of theplurality of customer images; receive a selection of a merchant item fordisplay; determine a virtual object corresponding to the selection of amerchant item; receive from the first computing device a positioningsignal corresponding to a desired location of the virtual object in aparticular 2D view of the customer environment; determine acorresponding first location in each of the plurality of customer imagesat which to overlay the virtual object based on the positioning signal,the corresponding first location corresponding to the desired locationin each respective different 2D view of the customer environment;determine a size and orientation of the virtual object in each of theplurality of customer images based on corresponding 3D feature data;cause an overlay of the virtual object, having the determined size andorientation, at the corresponding first location in each of theplurality of customer images to create a respective plurality ofaugmented images for display on a second computing device, wherein thesecond computing device is a separate device from the first computingdevice; receive a re-positioning signal indicative of a desiredre-positioned location of the virtual object in the particular 2D viewor a different 2D view of the customer environment; determine acorresponding second location in each of the plurality of customerimages at which to overlay the virtual object based on there-positioning signal, the corresponding second location correspondingto the desired re-positioned location in each respective different 2Dview of the customer environment; and update the overlay of the virtualobject to be positioned at the corresponding second location in each ofthe plurality of customer images.